Distributed multi-task learning provides significant advantages in multi-agent networks with heterogeneous data sources where agents aim to learn distinct but correlated models simultaneously. However, distributed algorithms for learning relatedness among tasks are not resilient in the presence of Byzantine agents. In this paper, we present an approach for Byzantine resilient distributed multi-task learning. We propose an efficient online weight assignment rule by measuring the accumulated loss using an agent's data and its neighbors' models. A small accumulated loss indicates a large similarity between the two tasks. In order to ensure the Byzantine resilience of the aggregation at a normal agent, we introduce a step for filtering out larger losses. We analyze the approach for convex models and show that normal agents converge resiliently towards their true targets. Further, an agent's learning performance using the proposed weight assignment rule is guaranteed to be at least as good as in the non-cooperative case as measured by the expected regret. Finally, we demonstrate the approach using three case studies, including regression and classification problems, and show that our method exhibits good empirical performance for non-convex models, such as convolutional neural networks.

Weaknesses: First, the scope of this paper is limited. Existing deep multi-task learning literatures are totally ignored. On the other hand, deep distributed learning is also not considered in this work, such as "Large Scale Distributed Deep Networks by Jeffrey Dean et al.", "FedNAS: Federated Deep Learning via Neural Architecture Search by Chaoyang He et al.". Both aspects should be incorporated into the paper to provide a whole picture of the development of distributed multi-task learning. Second, the proposed method is based on some strong convex assumption.

There was some disagreement in the initial reviews. Three reviewers were quite positive, noting that the intuitive algorithmic ideas, interesting results and good empirical evaluation. One reviewer was more negative, with some concerns regarding the scope of the approach (in particular due to the convex assumption) and the lack of discussion/evaluation on deep MTL models. After reading the author rebuttal and further discussion among reviewers, I consider that the restriction of the analysis to convex scenarios is acceptable given that this work appears to be the first one on Byzantine MTL and that the authors provide a basic experiment in the nonconvex setting. Therefore, the paper is accepted, but I ask the authors to add a detailed discussion of the nonconvex setting and encourage them to include more complete deep learning experiments in the final version.

Distributed multi-task learning provides significant advantages in multi-agent networks with heterogeneous data sources where agents aim to learn distinct but correlated models simultaneously. However, distributed algorithms for learning relatedness among tasks are not resilient in the presence of Byzantine agents. In this paper, we present an approach for Byzantine resilient distributed multi-task learning. We propose an efficient online weight assignment rule by measuring the accumulated loss using an agent's data and its neighbors' models. A small accumulated loss indicates a large similarity between the two tasks.

Distributed training of deep neural networks faces three critical challenges: privacy preservation, communication efficiency, and robustness to fault and adversarial behaviors. Although significant research efforts have been devoted to addressing these challenges independently, their synthesis remains less explored. In this paper, we propose TernaryVote, which combines a ternary compressor and the majority vote mechanism to realize differential privacy, gradient compression, and Byzantine resilience simultaneously. We theoretically quantify the privacy guarantee through the lens of the emerging f-differential privacy (DP) and the Byzantine resilience of the proposed algorithm. Particularly, in terms of privacy guarantees, compared to the existing sign-based approach StoSign, the proposed method improves the dimension dependence on the gradient size and enjoys privacy amplification by mini-batch sampling while ensuring a comparable convergence rate. We also prove that TernaryVote is robust when less than 50% of workers are blind attackers, which matches that of SIGNSGD with majority vote. Extensive experimental results validate the effectiveness of the proposed algorithm.